How to run wandb offline

1. Create a python environment.
2. Activate python environment with wandb.
3. Install wandb library pip install --no-index wandb.
4. Run wandb login to set up your login key. Follow the prompt and paste your api keys (can be found in user settings).
5. Add export WANDB_MODE="offline" before python or accelarate launch line in the shell script.
6. To run wandb offline with your python script, add os.environ["WANDB_MODE"]="offline" to your py script.
7. Input your project name and run name in wandb.init(). For an example, wandb.init( project='wandbtest', name='test', entity='lence_ubc',)
8. Use wandb.log() to log your metrics. For an example, wandb.log({"loss": loss})
9. Run your your python script. When your script is done, your logs will be stored inside a new folder./wandb/offline-run... located in the same directory as your python script.
10. In the terminal, run wandb init in your directory to set the project name.
11. Run wandb sync ./wandb/offline-run* to push all the saved metrics to the cloud service.

NOTE If you run your code on SLURM_TMPDIR, you will need to zip the wandb folder zip -qr wandb.zip ./wandb and copy it to the project space before doing the last two steps.